Method and system enabling real time mixing of synthetic images and video images by a user

ABSTRACT

The invention concerns a method and a system for: (i) producing in a computer processing unit a flow of synthetic images, and (ii) tracing a scene by creating visual interactions between the synthetic image flow and at least one video image flow. The computer processing unit comprises: a motherboard, a graphics board for scene rendering and display, including a processor for accelerating 2D/3D processing, a work buffer and a texture memory, an acquisition means for acquiring in real time video images, in a video buffer. The specific rendering of the scene is carried out: by recopying the video buffer in a memory zone of the graphics board, by tracing the synthetic images in the work buffer.

PREAMBLE OF THE DESCRIPTION

Domain involved, problem in question.

The present invention concerns a method and a system permitting a userto mix synthetic images and video images in real time.

More precisely, the subject matter of the invention is a method and asystem enabling a user:

(i) to produce in a data processing unit, composed of standardelectronic circuits, a flow of synthetic images, and

(ii) to trace a scene by creating visual interactions between the flowof synthetic images and at least one flow of video images.

The solution of this problem presented by the invention permitsimproving the quality of the scene and to shorten the processing timewithout employing proprietary and expensive data processing units.

Prior Art

An equipment according to the prior art is represented in FIG. 1 whichpermits the achievement of the improved reality. This equipmentincludes;

-   -   a graphic work station STG,    -   a rack of the “chroma keyer” type RCK,    -   a rack of the “image delay” type RRI.

The legends C and SV indicate respectively a camera and a video output.

The graphic work station STG is used for the generation only ofsynthetic images. The graphic work statio STG is developed specificallyor is a Silicon Graphics™ station. The synthetic images are generated ona uniform color background (key color).

The “chroma keyer” type rack RCK is used for mixing the video image fromthe camera (C) with the synthetic image generated by the graphic stationSTG. The key color is replaced by the video image.

The “image delay” type rack RRI is used to compensate for the processingtime of the image generator.

Drawbacks of the prior technology:

Equipment of the prior art present problems in regard to price and imagequalities and functionality.

Problems in regard to price:

The cost price of the equipment of the prior art is very high. In fact,the “image delay” rack RRI is a professional rack, the graphic stationSTG is a professional station. The previous solution according to theprior art is not carriable(it is made up of three separate modules:image generator/image delay/chroma key).

Problems in regard to image quality and functionality:

Visual interactivity between the video and the synthetic images islimited. In fact, at the output from the “chroma keyer” RCK the video isinserted in place of the key color. It is not possible to havereflections of the video on the synthetic images or vice versa. Thevideo cannot be treated as a texture. For example, it cannot havereflection effects between the real images (video images) and thesynthetic objects. The very principle of the “chroma key” can produceartefacts; it limits selection in the color palette of synthetic images,and makes the rendition of synthetic objects that have transparenciesproblematical.

The performance of specific graphic stations is quickly exceeded by theperformance of consumer graphic boards which are ever more powerful(graphic power of consumer equipment is doubled about every six months).

The previous solution is expensive in terms of maintenance (the threeunits composing it involve a specific maintenance contract).

The previous solution cannot operate in a portable computer; it is notmobile.

The previous solution is constraining as regards time involved inadjusting all its parts.

On processing of the video image is possible:

-   -   the video cannot be used to texturize locally any element of the        virtual scene,    -   the video cannot be shaped (for example the correction of        optical distortions is impossible),    -   no processing can be performed on the video (for example,        processing of the chroma key type).

SOLUTION

Process

The data processing unit according to the invention, particularly of the“PC computer” type, comprises:

-   -   a mother board    -   a graphic board for rendering and displaying the scene.

The meaning of scene in the sense of this invention is set forth belowin connection with FIG. 2.

The graphic board for rendering and displaying the scene comprises aprocessor for accelerating 1D and 3D processings, a working buffer “backbuffer” and a texture memory.

The data processing equipment also includes an acquisition means makingit possible to acquire video images in real time in a video buffer. Theacquisition means appears particularly in the form:

-   -   of a video acquisition integrated into the graphic board, and/or    -   of a video acquisition integrated into the mother board, and/or    -   of a video acquisition via a dedicated acquisition card (for        example a Pinnacle™ card known by the name of PCTV).

The method according to the invention includes the step of performing aspecific rendering of the scene,

-   -   by copying the video buffer, upon each rendering of the scene,        in a memory area of the graphic board,    -   by tracing the synthetic images in the said working buffer of        the graphic board.

According to a first variant embodiment of the invention, to carry out aspecific rendering, preferably the method includes the following steps:

-   -   the step of copying the video buffer into the working buffer        after each rendering of the scene,    -   the step of tracing the synthetic images in the working buffer.

In the case of this first variant embodiment, when the video bufferincludes interlaced video lines, to copy the video buffer, upon eachrendering of the scene, into the working buffer, preferably theprocedure includes:

-   -   the step of copying the even video lines, in a first rendering,        then    -   the step of copying the odd video lines, in the following        rendering.

According to a second variant embodiment of the invention, to carry outa specific rendering the method includes, preferably, an initializationstep prior to the specific rendering, consisting of creating a dedicatedtexture in the texture memory of the graphic board. The dedicatedtexture is of the size of the video buffer. The texture is dedicated forcopying the video buffer into the texture memory.

In the case of this second variant embodiment, the method furthermoreincludes:

-   -   the step of copying the video buffer into the dedicated texture,    -   the step of completely retracing the scene, using the dedicated        texture to texture some of the polygons of the scene.

In the case of this second variant embodiment, when the video buffercontains interlaced video lines, to copy the video buffer, upon eachrendering of the scene, into the dedicated texture, preferably accordingto the invention the method comprises:

-   -   the step of copying the even video lines in a first rendering,        then    -   the step of copying the odd video lines in the following        rendering,    -   the step of applying a translation of the video buffer of more        or less one half-line to the rendering of the scene.

This translation of the video buffer is performed:

-   -   either by modifying the texture coordinates of the dedicated        texture,    -   or by modifying the coordinates of the polygons textured by the        dedicated texture.

The image quality is thus improved.

In the case of the first variant embodiment as in the case of the secondvariant embodiment, it is possible to copy the video buffer into thememory area of the graphic board by proceeding in two different ways.

In the case of a first way, the means of acquisition includes a driverhaving a new video buffer for each new video image. Preferably accordingto the invention, in the case of this first way, the rendering of thescene is performed in synchronism with the presentation of each newvideo buffer.

In the case of the second way, the acquisition means also includes adriver presenting a new video buffer for each new video image. However,in the case of this second manner of proceeding, preferably according tothe invention the method comprises the following steps:

-   -   the step of filling a buffer memory with each new video buffer        by employing a unit of performance (“thread”) of video capture,    -   the step of copying one of the video buffers contained in the        buffer memory to the texture memory of the graphic board,    -   the step of performing the scene rendition asynchronously with        respect to the presentation of each new video buffer.

The method according to the invention permits improved graphicfunctionality in the designing of a scene mixing a video image with asynthetic image.

Anti-Aliasing

Preferably, according to the invention, the method includes the step ofapplying an anti-aliasing function in the designing of the scene. Theideas of anti-aliasing function will be described below.

Transparency

Preferably, according to the invention, the method includes the step ofapplying a transparency function during the designing of the scene. Theideas of transparency function will be described below.

Correction of Optical Distortion

Preferably, according to the invention, the method includes the step ofapplying non-linear distortions to the video buffer by texturing thepolygons of the scene by means of the dedicated texture.

It is thus possible to correct optical distortion of the video image.

Real-time Processing of Video Images

Preferably, according to the invention, the method includes the step ofapplying pixel shaders functions to the video buffer, permittingespecially a chroma key type processing. The idea of pixel shaders andprocessing of the chroma key type will be described below.

Environment Mapping

Preferably, according to the invention, the data processing unitincludes two means of acquisition, each having a video buffer. Themethod comprises the following steps:

-   -   the step of copying the video buffer of one of the acquisition        means into a second dedicated texture of the graphic board,    -   the step of texturing at least a portion of the scene by using        the second dedicated texture.

It is thus possible to obtain real reflections on the synthesis objects.

Video Delay

Preferably, according to the invention, the method includes the step ofslowing the copy of one of the video buffers contained in a buffermemory toward the texture memory of the graphic board.

It is thus possible to delay the display of the video images withrespect to the display of the synthetic images.

System

The invention likewise concerns a system enabling a user:

(i) to produce in a data processing unit, composed of standardelectronic circuits, a flow of synthetic images, and

(ii) to design a scene, creating visual interaction between thesynthetic image flow and at least one flow of video images.

The solution of this problem makes it possible to improve the quality ofthe scene and to reduce the processing time, without making use ofproprietary and expensive data processing units.

The data processing unit according to the invention includes aprocessor, particularly of the common “PC” type. This processorcomprises:

-   -   a mother board    -   a graphic board for rendering and displaying the scene.

The rendering and displaying graphic board contains a processor foraccelerating 2D/3D processings, a working “back buffer” and a texturememory.

The data processing equipment includes an acquisition means allowing theacquisition in real time of video images in a video buffer. Theacquisition means appears particularly in the form:

-   -   of a video acquisition means integrated into the graphic board,        and/or    -   a video acquisition means integrated into the mother board,        and/or    -   a means of video acquisition via a dedicated acquisition card.

To perform a specific rendition of the scene:

-   -   the mother board furthermore contains a data processing means        making it possible, upon each rendition of the scene, to copy        the video buffer into a memory zone of the graphic board,    -   the 2D/3D processing accelerating processor includes tracing        means for tracing designing the synthetic images in a memory        area of the graphic board.

In a first variant embodiment, preferably according to the invention, tocarry out a specific portrayal rendition of the scene:

-   -   the data processing means comprise first copying means for        copying, at each portrayal of the scene, the buffer video into        the working buffer,    -   the 2D/3D processing accelerating processor comprises first        computing means for tracing the synthetic images into the        working buffer.

In the case of this first variant embodiment, when the video buffercontains interlaced video lines, then preferably according to theinvention, the first means for copying from the buffer video into theworking buffer comprise:

-   -   first means for selecting and copying the even video lines, in a        first rendition, then    -   second means for selecting and copying the odd video lines in        the following rendition.

In a second variant embodiment, to perform a specific rendition,preferably according to the invention, the data processing meanscomprise initialization means creating, prior to the specific rendition,a dedicated texture in the texture memory of the graphic board. Thededicated texture has the size of the video buffer. In the case of thissecond variant embodiment, the data processing means furthermorecomprise second copying means for copying the video buffer into thededicated texture,

-   -   the 2D/3D processing acceleration processor comprises second        computing means for completely tracing the scene, using the        dedicated texture for texturing certain polygons of the scene.

In the case of this second variant embodiment, when the video buffercontains interlaced video lines, preferably according to the invention,the second means for copying from the video buffer into the dedicatedtexture comprise:

-   -   third selection and copying means for the even video lines in a        first rendition, then    -   fourth selection and copying means for the odd video lines in        the following rendition.

The 2D/3D processing acceleration processor comprises third computationmeans in order to apply a translation from the video buffer of more orless one half-line in the rendition of the scene.

This translation from the video buffer is performed:

-   -   either by modifying the texture coordinates of the dedicated        texture,    -   or by modifying the coordinates of the polygons textured by the        dedicated texture.

Thus the image quality is improved.

In the case of the first variant embodiment, as in the case of thesecond variant embodiment, it is possible to copy the video buffer intwo different ways:

In the case of the first way of proceeding, the acquisition meanscomprises a driver having a new video buffer for each new video image.Preferably, according to the invention, in the case of this first way ofproceeding, the data processing means perform the rendition of the scenein synchronism with the presentation of each new video buffer.

In the case of a second way of proceeding, the acquisition meanscomprises a driver having a new video buffer for each new video image.Preferably, according to the invention, in the case of this second wayof proceeding, the said data processing means comprise:

-   -   transfer means for filling a buffer memory with each new video        buffer, using a video capture performance unit (“thread”),    -   third copying means for copying a video buffer contained in the        said buffer memory to the texture memory of the graphic board.

The data processing means perform the rendering of the said scene out ofsynchronism with respect to the presentation of each new video buffer.

The system according to the invention permits graphic functionalitieswhen a scene is being scanned by mixing a video image with a synthesisimage.

Anti-Aliasing

Preferably, according to the invention, the data processing means makeit possible to apply an anti-aliasing during the designing of the scene.

Transparency

Preferably, according to the invention, the data processing means permitapplying a transparency function during the scanning of the scene.

Correction of Optical Distortion

Preferably, according to the invention, the data processing means permitapplying non-linear distortion to the video buffer by texturing polygonsof the scene by means of the dedicated texture.

It is thus possible to correct optical distortions of the video image.

Real-Time Processing of Video Images

Preferably, according to the invention, the data processing means permitapplying pixel shader functions, permitting in particular a processingof the chroma key type.

Environment Mapping

Preferably, according to the invention, the data processing unitincludes two acquisition means each having a video buffer. The dataprocessing means permit copying the video buffer of one of theacquisition means into a second dedicated texture of the graphic board.

The 2D/3D processing accelerating processor permits texturing at least aportion of the scene by using the second dedicated texture.

It is thus possible to obtain real reflections on the synthetic objects.

Video Delay

Preferably, according to the invention, the data processing means permitslowing the copy of one of the video buffers contained in a buffermemory toward the texture memory of the graphic board.

It is thus possible to delay the display of the video images withrespect to the display of the synthetic images.

DETAILED DESCRIPTION

Other characteristics and advantages of the invention will appear fromreading the description of a variant embodiment of the invention, givenby way of suggestive but not limiting example, and of

FIG. 1 representing equipment of the prior art making it possible tomake improvements to reality

FIG. 2 representing an overall plan of a variant embodiment of a systemaccording to the invention,

FIG. 3 schematically showing the data processing means 26 in the case ofanother Variant embodiment,

FIG. 4 representing schematically the 2D/3D processing accelerationprocessor 9 in the case of the variant embodiment shown in FIG. 3,

FIG. 5 representing schematically a variant embodiment in the case inwhich the acquisition means 12 appears in the form of a videoacquisition integrated into the graphic board 8,

FIG. 6 representing schematically a variant embodiment in the case wherethe acquisition means 12 is in the form of a video acquisitionintegrated into the motherboard 7,

FIG. 7 schematically representing a variant embodiment in a case wherethe acquisition means 12 appears in the form of a video acquisition byway of a dedicated acquisition card,

FIG. 8 schematically representing the copy algorithm of the video buffer14 in the video output buffer (back buffer) 40 of the graphic board,

FIG. 9 schematically representing the copy algorithm of the video buffer14 in the texture memory 11 of the graphic board.

FIG. 1 has been described above in the presentation of the prior art.Now FIG. 2 will be described, which represents an overall plan of avariant embodiment of a system according to the invention.

In FIG. 2 there is represented a data processing unit 2 comprising:

-   -   a graphic board 8,    -   a motherboard 7    -   acquisition means 12 receiving a flow 5 of video images 13,    -   a hard disk 42 containing a scene 4.

The graphic board 8 comprises:

-   -   a 2D/3D processing acceleration processor 9 comprising first        computing means 29,    -   a work buffer 10 receiving the flow 3 of synthetic images 16        from the 2D/3D processing acceleration processor 9,    -   a texture memory 11 comprising a dedicated texture 19,    -   a video output 39,    -   a video output buffer (back buffer) 40.

The motherboard 7 comprises:

-   -   data processing means 26 comprising first copy means 27        containing first selection and copy means 30, [and] second        selection and copy means 31,    -   a central memory 41 containing a scene 4 and polygons 20,    -   a buffer memory 23 comprising video buffers 14 containing even        video lines 17 and odd video lines 18,    -   a driver 22,    -   an execution unit 24.

In another variant embodiment, the data processing means 26 includeinitialization means 32, second copy means 33 comprising the third meansof selection and copy 35, [and] fourth selection and copy means 35 (seeFIG. 3). In the case of this variant embodiment, the 2D/3D processingacceleration processor 9 includes second computing means 34 (see FIG.4),

The acquisition means 12 numbers the video signal 13 corresponding tothe video images of the flow 5 (the video signal can be in the analog ornumerical standard). The mother board receives this numbered flow in avideo buffer 14 composed of pixels and stored in its memory. In thedescription, the video buffer 14 is also called the pixel buffer.

A 3D scene 4 is a combination of elements:

-   -   3D objects composed of textures (2D image) and polygons 20 (3D        geometry).    -   Light sources    -   Virtual cameras.

The 3D scene 4 is stored on the hard disk 42 and is recopied into thecentral memory 41 of the motherboard 3.

To calculate a synthesis image 16, the motherboard 7 copies the videobuffer 14 and the textures of the 3D scene in the texture memory 11 ofthe graphic board 8 and sends to the latter all of the polygons 20 to bedisplayed, the light sources, [and] the camera. The specialized circuits(the 2D/3D processing accelerating processors 9) compute from these 3Ddata an image 2D in the work buffer 10 (back buffer). The content of thework buffer 10 is then transferred into the video output buffer 40(front buffer) and the video output 39 to be used by the display device(television, monitor, projector) to display the image.

The video buffer 14 is recopied into the dedicated texture 19 of thetexture memory 11. Since the dedicated texture 19 is part of the scene,it is possible to display its content according to the needs of theuser. The following displays are mentioned as non-restrictive examples:

-   -   display of the dedicated texture 19 in the background to obtain        a “full screen” video.    -   display of the dedicated texture 19 on a synthetic object        included in the scene 4 to enrich the latter with a video in        real time.

Texturization is a technique that consists in drawing a polygon 20 in 3Dwith a texture (2D image) superimposed. For this purpose, each apex 3Dis assigned a polygon 20 of 3D coordinates (texture coordinates) in thetexture. It is possible to use several textures on the same polygon 20.A set of texture coordinates for each texture is associated with theapexes.

The texturization can be done with any type of 2D image, and thereforewith the video buffer 14 which is a 2D image. A set of polygons 20,texturized with the video buffer 14, is treated like any object involvedin the computing of the synthetic image 16.

A description will now be given of FIG. 5 schematically represented avariant embodiment in a case where the acquisition means 12 appears inthe form of a video acquisition integrated into the graphic board 8.

The data processing unit 2 contains a graphic board 8 equipped with anacquisition means 12. The video image 13 is acquired in real time by theacquisition means 12 and then recopied into the video buffer 14. Thevideo buffer 14 is then recopied in two possible ways:

-   -   either to the work buffer 10,    -   or to the texture memory 11.

After this copy, the 2D/3D processing acceleration processor 9 performsa rendering of the scene 4 in the work buffer 10. The work buffer 10 istransferred into the video output buffer 40 which makes it possible toupdate the video output 39.

Now a description will be given of FIG. 6 which represents schematicallya variant embodiment in a case where the acquisition means 12 appears inthe form of a video acquisition integrated into the motherboard 7.

The data processing unit 2 contains a motherboard 7 equipped with anacquisition means 12. The data processing unit 2 likewise contains agraphic board 8. The video image 13 is acquired in real time by theacquisition means 12, then recopied into the video buffer 14. The videobuffer 14 is then recopied in two possible ways:

-   -   either to the work buffer 10,    -   or to the texture memory 11.

After this copy, the 2D/3D processing acceleration processor 9 performsa rendering of the scene 4 in the work buffer 10. The work buffer 10 istransferred to the video output buffer 40 permitting the updating of thevideo output 39.

A description will now be given of FIG. 7 which represents schematicallya variant embodiment in the case where the acquisition means 12 appearsin the form of a video acquisition via a dedicated acquisition card.

The data processing unit 2 contains an acquisition means 12 in the formof an acquisition card connected to the motherboard 7 (for example, aPinnacle™ PCTV card. The data processing unit 2 likewise contains agraphic board 8. The video image 13 is acquired in real time by theacquisition means 12, then recopied into the video buffer 14. The videobuffer 14 is then recopied in two possible ways:

-   -   either to the work buffer 10,    -   or to the texture memory 11.

After this copy the 2D/3D processing accelerating processor 9 of thegraphic board performs a rendering of the scene 4 in the work buffer 10.The work buffer 10 is transferred into the video output buffer 40permitting the updating of the video output 39.

A description will now be given of FIG. 8 which schematically representsin block form the copying of the video buffer 14 into the video outputbuffer (back buffer) 40 of the graphic board. The blocks have thefollowing meaning:

Block 48:

START

Block 50:

Initialization of the acquisition means 12:

Change to acquisition mode.

Definition of the video input (e.g.: Svideo/PAL/NTSC

Starting of the video acquisition.

Block 52:

Initialization of the 2D/3D processing acceleration processor 9:

Definition of the type of output (e.g.: VGA or PAL or NTSC or DVI).

Definition of graphic mode (full screen or window).

Definition of the graphic resolution (e.g.: 800 by 600 at 32 bits perpixel).

Definitial of the mode Z buffer (init. of 3D acceleration).

Change to double buffer mode (creation of video output 40 and workbuffer 10).

Block 54:

Change of scene 4:

Loading of all 3D models from the hard disk 42.

Block 56:

RAZ of the Zbuffer.

RAZ of the work buffer 10 (optional).

Block 58:

Copying of video buffer 14 coming from the acquisition means 12 into thework buffer 10.

Block 60:

Tracing of scene 4 into the work buffer 10 performed by the 2D/3Cprocessing acceleration processor 9.

Updating of video output buffer 40

Block 62:

End program?

Block 64:

THE END.

A description will now be given of FIG. 9 schematically representing inblock form the copy algorithm of the video buffer 14 into the texturememory 11 of the graphic board.

The blocks have the following meaning:

Block 66:

START

Block 68:

Initialization of the acquisition means 12:

Change to acquisition mode

Definition of video input (e.g.: Svideo/PAL/NTSC)

Block 70:

Initialization of the 2D/3D processing acceleration processor 9,

Definition of type of output (e.g.: VGA or Pal or NTSC or DVI).

Definition of the graphic mode (full screen or window).

Definition of the graphic resolution (e.g.: 800 by 600 at 32 bits perpixel)

Definition of the Zbuffer mode (init. 3D acceleration).

Shift to double buffer mode (creation of the video output buffer 40 andwork buffer 10).

Initialization of the dedicated texture (creation of the dedicatedtexture 19 in the texture memory 11, the dedicated texture 19 at thesame number of lines and columns as the video capture buffer video 14)

Block 72:

Loading of scene 4

Loading of all 3D models from the hard drive 42.

Block 74:

RAZ of the Zbuffer.

RAZ of the work buffer (10) (optional).

Block 76:

Copy of the video buffer 14 from the acquisition means 12 in thededicated texture 19 included the texture memory 11.

Block 78:

Tracing of scene 4 into the working buffer 10

Utilization of textures, including those that can be made of a texture(e.g., environment mapping, traced internal combustion engine he workbuffer 10 as screen background, texturization of synthetic objects).

Updating of the video output buffer 40.

Block 80:

End of program?

Block 82:

END.

Description of the anti-aliasing:

Anti-aliasing is the filtering operation that permits reduction of thestair-steps on the edges of the polygons 20 due to the finite resolutionof the synthetic image 16.

When anti-aliasing is applied, the color of the pixels on the edges ofthe object in the foreground is obtained by an operation of filteringbetween the pixels of the background objects and the pixels of theobject in the foreground.

The invention makes it possible to treat a textured object with thevideo as an element of the same nature as the other synthetic objects:the anti-aliasing functions therefore are applied in tracing the entirescene.

Description of Transparency Effects

The synthetic objects can contain transparent surfaces. If such anobject is in the foreground in relation to a background object texturedwith a video, the texured video object will be seen through thesesurfaces.

Description of Non-Linear Distortion:

The method permits the application of non-linear distortion to thevideo, since the latter is stored in a texture mapped on polygons 20. Itis thus possible to distort the video image in real time to correct theoptical distortions of the video camera. The lines at the edges of theimage that appear to be curved can thus be straightened by pulling thevideo texture.

For this purpose a regular mesh of polygons 20 is created on which avideo texture is plated with texture coordinates describing a curve whenthe coordinates of the associated vertices evolve linearly.

Description of “pixel shader” Processings,

Particularly on the “Chroma Key.”

The final color of a pixel is a combination between the color of thissame pixel for the polygon 20 being traced and the color for thepolygons 20 already traced. The color of a pixel of a polygon is acombination between the value of this pixel in the texture or texturesassociated with these polygons and the illumination of the polygon.

The present-day graphic boards use a program (pixel shader) to performthis combination. This program is transferred into the graphic board 8like the other 3D objects.

The method of the invention pen-nits applying functions, known inthemselves, of the “pixel shader” type, to the video image. This makesit possible particularly to employ algorithms of the “chroma-key” typeon the video buffer 14, or other image processings.

The principle of the chroma key consists in replacing in an image allthe points of a given color with points from another video source.Sometimes this technique is called “blue screen.” A description isprovided below of a simple chroma-key algorithm. For each pixel of thevideo:

-   -   its color is entered in the space YUV,    -   the distance of this color is calculated in relation to a        reference color, ignoring the coordinate Y (the luminance),    -   if the distance is less than a predetermined threshold the pixel        is made transparent; otherwise it is displayed normally.

Description of Environment Mapping

Environment mapping is a technique that simulates reflective surfaceswithout using ray tracing (unusable in real time). The technique appliesa special texture containing an image of the scene surrounding theobject on the object itself. The result obtained is an approximation ofthe appearance of a reflective surface, without requiring too muchcomputing power.

The method of the invention, generalized to the use of two acquisitionmeans 12 (two acquisition cards) permits using one of the twoacquisition means 12 (one of the two acquisition cards) to copy thebuffer video 14 associated with this card in the memory of a texture 11which will serve as a mapping of the environment when objects of thetexturized scene are traced with this texture. This particularly permitsobtaining real reflections on these synthetic objects.

The other acquisition card is used to copy its video buffer 14 to thememory of a texture displayed in the background of the synthetic objectson which there are reflections.

Description of a Buffer Memory 23

The buffer memory method permits slowing the display of the videoimages.

N video images are stored in memory.

The buffer memory method permits slowing the display of video images.The buffer memory 23 operates on the following principle: the buffermemory 23 can store N video buffers 14. These video buffers 14correspond to the video images 13 acquired by the acquisition means 12during the cycles N, (N-2), (N-2), (N-3), etc . . . In cycle N, themethod can decide to use the video buffer 14 acquired at the cycle(N-2), which in this case results in a slowing of the video imagesdisplayed by the video output 39 by 3 cycles. Generally, by thisprocess, up to N delayed images can be introduced.

Advantages of the Claimed Solution over the Prior Art

Now the advantages of the method and system of the invention over theprior art will be explained:

-   -   The method and the system of the invention permit an improvement        of the quality of the images: number of polygons, shadowing,        texturizing, transparency, reflections and anti-aliasing.    -   The method and the system of the invention make it possible to        achieve in real time the effects of video texturization, that        is, the texturization of synthetic objects, not with a single        static image, but with a live video.    -   The method and the system of the invention make it possible to        achieve dynamic environment mapping effects, that is, the        possibility of surrounding the synthetic objects with an        environment map that is not static but is the video itself. This        opens the door to surprising effects, particularly when the        synthetic objects have as reflective a surface as a mirror.    -   The method and the system of the invention permit achieving        everything that is possible to do classically with static        textures, but now with one or several live video sources.    -   The method and the system of the invention permit carrying out a        video image processing by the technology of pixel shaders        (particularly processing of the chroma key type.    -   The method and the system of the invention permit slowing video        images without an external rack.    -   The method and the system of the invention permit a reduction of        the system's response time since all of the processings are        internal in the same machine (the processing times of several        processing units do not accumulate as internal combustion engine        he prior art).    -   The method and the system of the invention permit the use of        materials produced in large quantity at low cost.    -   The method and the system of the invention do not require        electronic proprietary developments (hardware): they permit ease        of maintenance (mother boards of the generally available PC,        consumer graphic boards, consumer video acquisition cards).    -   The method and the system of the invention can work with a        single computer (standard PC case), portable computer, or        industrial rack PC).    -   The method and the system of the invention can operate in real        time: the video images are generated directly. It is thus        possible to eliminate the post-production costs (no time of        several hours spent for a few seconds of video images).    -   The method and the system of the invention can be shipped. The        method and the system can operate on compact industrial PC        boards (military or naval or dangerous areas).    -   The method and the system of the invention can operate on        hardened material (for industry, military).    -   The method and the system of the invention can follow the        development of performances of accelerating graphic boards with        the minor task of updating the software: use of standard graphic        libraries (DirectX or Open    -   The method and the system of the invention can easily be moved        (mobility) and quickly installed where it will be used.    -   The method and the system of the invention can run on a portable        computer equipped with a video input (e.g.: DV input).

Industrial and commercial applications of the method and systemaccording to the invention.

Industrial applications internal combustion engine he audiovisual field(motion pictures and video).

The method and the system of the invention can particularly be used foraudiovisual productions shot on line or under on-line conditions:

-   -   virtual sets in real time (display of presenters in virtual        settings).    -   special effects in real time (e.g., particles, explosions,        distortions in real time)

The method and the system of the invention can be used especially in theadvertising field (e.g., virtual products in real settings).

The method and the system of the invention can be used in order tooverlay virtual objects in a telecast and to create interaction with alive presenter (talkshows, sporting events, television sales, news,weather forecasts, medical broadcasts).

The method and the system of the invention can be used in doing“avatars,” virtual personalities in real settings in real time.

The method and the system of the invention can be used in makingreal-time visualizations for motion pictures (control of special effectsat the site of the shooting).

INDUSTRIAL APPLICATIONS IN THE INDUSTRIAL FIELD

The method and the system of the invention can be used in themanufacturing field (enrichment with real images to guide manualoperations and inform the manufacturer during production).

The method and the system of the invention can be used in themaintenance field (enrichment with real images with supplementalinformation to optimize maintenance schedules, real-time guide foremergency repairs).

The method and the system of the invention can be used internalcombustion engine the military (virtual targets, virtual threats,explosions, infantry, vehicles).

The method and the system of the invention can be used in simulations(simulated atmospheric or environmental phenomena mixed with real props,testing of virtual prototypes, virtual vehicles).

The method and the system of the invention can be used in the field ofvideo games (real players visualized in a video game, or virtualelements such as persons, vehicles integrated into real images).

The method and the system of the invention can be used:

-   -   in the automotive field (virtual vehicles on real roads).    -   in the field of architecture (visualization of virtual buildings        in real landscapes).    -   in the field of real-time driving: display of data to direct a        pilot or a conductor.

Applications in the Marketing or Communication Field

The method and the system of the invention can be used in conductingmarket studies: display of products not yet actually made, in realsituations (e.g.: new mobile GSM, new vehicle).

The method and the system of the invention can be used to makeinteractive images in shops or business centers.

Industrial applications in the field of information technology.

The method and the system of the invention can be used in videoconferences: visiophone with possibility of inserting and interactingwith virtual objects.

INDUSTRIAL APPLICATIONS IN THE LEISURE FIELD

The method and the system of the invention can be used in amusementparks: visual animations in real time (e.g.: interactive mirrors).

1-26. (canceled)
 27. A method for enabling real-time mixing of syntheticimages and video images by a user, comprising the steps of: producing aflow of synthetic images using a data processing unit comprising amotherboard; a graphic board for rendering and displaying said scene,and comprising a 2D/3D processing acceleration processor, and a memoryzone comprising a work buffer and a texture memory; and acquisitionmeans for acquiring, in real time, video images in a video buffer; andtracing a scene by creating visual interactions between said flow ofsynthetic images and at least a flow of video images; and performing aspecific rendition of said scene by copying, upon each rendering of saidscene, said video buffer into said memory zone of said graphic board;and tracing said synthetic images into said memory zone of the graphicboard, thereby improving the quality of said scene and reducing theprocessing time without employing expensive proprietary data processingunits.
 28. The method of claim 27, wherein the step of performingcomprises the steps of: copying, upon each rendering of said scene, saidvideo buffer into the said work buffer; and tracing said syntheticimages into said work buffer.
 29. The method of claim 28, wherein saidvideo buffer comprises interlaced video lines; and further comprisingthe steps of: copying even video lines of said video buffer, upon afirst rendition of said scene, into said work buffer; and copying oddvideo lines of said video buffer, upon the following rendition of saidscene, into said work buffer.
 30. The method according of claim 27,further comprising the steps of: initializing prior to the specificrendition by providing a dedicated texture in said texture memory ofsaid graphic board, wherein said dedicated texture has the size of saidvideo buffer and is dedicated to copying said video buffer into saidtexture memory; and copying said video buffer into said dedicatedtexture and tracing said scene completely, using said dedicated textureto texture some of polygons of the said scene.
 31. The method of claim30, wherein said video buffer comprises interlaced video lines; andfurther comprising the steps of: copying even video lines of said videobuffer, at the time of a first rendering of said scene, into saiddedicated texture; and copying odd video lines of said video buffer, atthe time of the following rendition of said scene, into said dedicatedtexture; and applying a translation from said video buffer of about onehalf video line into the rendition of said scene either by modifyingtexture coordinates of said dedicated texture or the coordinates of thepolygons textured by said dedicated texture, thereby improving the imagequality.
 32. The method of claim 27, wherein said acquisition meanscomprises a driver having a new video buffer for each new video image;and wherein the step of performing renders said scene in synch withpresentation of each new video buffer.
 33. The method of claim 27,wherein said acquisition means comprises a driver having a new videobuffer for each new video image; and further comprising the steps of:filling a buffer memory with each new video buffer, employing a videocapture performance unit; copying one of the new video buffers containedin said buffer memory into said texture memory of said graphic board;and asynchronously performing the rendition of the said scene inrelation to presentation of each new video buffer.
 34. The method ofclaim 27, further comprising the step of applying an anti-aliasingfunction in the tracing of said scene.
 35. The method of claim 27,further comprising the step of applying a transparency function in thetracing of said scene.
 36. The method of claim 27, further comprisingthe step of applying nonlinear distortions to said video buffer whentexturing polygons of said scene using a dedicated texture, therebycorrecting optical distortions of the video image.
 37. The method ofclaim 27, further comprising the step of applying pixel shader functionsto said video buffer, thereby permitting the processing of chroma keytype.
 38. The method of claim 30, the said data processing unitcomprising one or more acquisition means, each having a video buffer;and further comprising the steps of: copying the video buffer of one ofsaid acquisition means into a second dedicated texture of said graphicboard; and texturing at least in part said scene using said seconddedicated texture, thereby obtaining real reflections on syntheticobjects.
 39. The method of claim 33, further comprising the step ofslowing the copying of one of the video buffers contained in a buffermemory into the said texture memory of said graphic board, therebyslowing the display of video images with respect to the display of thesynthetic images.
 40. System for enabling real-time mixing of syntheticimages and video images by a user, comprising: a data processing unitfor producing a flow of synthetic images and tracing a scene by creatingvisual interactions between said flow of synthetic images and at least aflow of video images, said data processing unit comprising: amotherboard comprising a data processor; a graphic board for renderingand displaying said scene; wherein said graphic board comprises a 2D/3Dprocessing acceleration processor comprising a tracing device, and amemory zone comprising a work buffer and a texture memory; and anacquisition device for acquiring, in real time, video images in a videobuffer; and wherein said data processing means is operable to perform aspecific rendition of said scene; wherein said data processor isoperable to copy, upon each rendition of said scene, said video bufferinto said memory zone of said graphic board; and wherein said tracingdevice is operable to trace said synthetic images into said memory zoneof the graphic board, thereby improving the quality of said scene andreducing the processing time without employing expensive proprietarydata processing units.
 41. The system of claim 40, wherein said dataprocessing means comprises first copying device for copying, upon eachrendition of said scene, said video buffer into said work buffer; andwherein said 2D/3D processing acceleration processor comprises a firstcomputing device for tracing said synthetic images into the said workbuffer.
 42. The system of claim 41, wherein said video buffer comprisesinterlaced video lines; and wherein said first copying device comprisesa first selection device for selecting and copying even video lines ofsaid video buffer at the time of the first rendering, and a secondselection device for selecting and copying odd video lines of said videobuffer at the time of the next rendering.
 43. The system of claim 40,wherein said data processing means comprises: a texture device forproviding, prior to the specific rendition, a dedicated texture in saidtexture memory of said graphic board, said dedicated texture having thesize of the said video buffer; and second copying device for copyingsaid video buffer into said dedicated texture to carry out a specificrendition; and wherein said 2D/3D processing accelerating processorcomprises a second computing device for completely tracing said sceneusing said dedicated texture to texture polygons of said scene.
 44. Thesystem of claim 43, wherein said video buffer comprises interlaced videolines; wherein said second copy means comprises: a third selectiondevice for selecting and copying said even video lines in a firstrendition; a fourth selection device for selecting and copying said oddvideo lines in the following rendition; a computing device for applyinga translation from said video buffer of about one half video line in therendition of said scene either by modifying texture coordinates of saiddedicated texture or the coordinates of polygons textured by saiddedicated texture, thereby improving the image quality.
 45. The systemof claim 40, wherein said acquisition device comprises a driver having anew video buffer for each new video image; and wherein said dataprocessor synchronously performs the rendition of said scene withpresentation of each new video buffer.
 46. The system of claim 40,wherein said acquisition device comprises a driver presenting a newvideo buffer for each new video image; and wherein said data processorcomprising: a transfer device for filling a buffer memory with each newvideo buffer, employing a unit of execution or thread of video capture,and a third copying device for copying one of the video bufferscontained in said buffer memory into said texture memory of graphicboard; and wherein said data processor is operable to asynchronouslyperform the rendition of said scene with respect to the presentation ofeach new video buffer.
 47. The system of claim 40, wherein said dataprocessor is operable to apply an anti-aliasing function when said sceneis being traced.
 48. The system of claim 40, wherein said data processoris operable to apply a transparency function when said scene is beingtraced.
 49. The system of claim 40, wherein said data processor isoperable to apply non-linear distortion to said video buffer whentexturing polygons of said scene using a dedicated texture, therebycorrecting optical distortions of the video image.
 50. The system ofclaim 40, wherein said data processor is operable to apply pixel shaderfunctions to said video buffer, thereby permitting the processing of thechroma key type.
 51. The system of claim 40, wherein said dataprocessing unit comprises one or more acquisition devices; wherein saiddata processor is operable to copy said video buffer from one of saidacquisition devices into a second dedicated texture of said graphicboard, and wherein said 2D/3D processing acceleration processor isoperable to texturize said scene at least in part by using said seconddedicated texture, thereby obtaining real reflections on syntheticobjects.
 52. The system of claim 40, wherein said data processor isoperable to slow the copy of one of said video buffers contained in abuffer memory into said texture memory of said graphic board (8),thereby slowing the display of the video images with respect to thedisplay of the synthetic images.